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This study presents the findings of a quasi-experimental feasibility study examining the Tertiary First Step 
intervention, an enhanced version of the First Step to Success early intervention program. Tertiary First 
Step was developed to engage families more effectively and influence and improve parenting practices for 
children having challenging behavior. Process (fidelity, dosage, and social validity) and outcome data were 
collected for all participants in the Tertiary First Step condition (N = 33). Parent- and teacher-reported 
outcomes were collected for the comparison condition (N= 22). Process data suggest the intervention was 
implemented with fidelity, and that teachers, parents, and coaches perceived the intervention as socially 
valid. This study presents the first empirical examination of the Tertiary Frist Step variation. The outcomes 
provide compelling evidence that the Tertiary First Step intervention is promising for improving student 
outcomes on social-behavioral indices, decreasing problem behavior, and improving academic engaged 
time. 
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Introduction 

Successfully navigating the social and behavioral expectations 
of today’s schools and classrooms is a challenging undertaking 
for young children. This task requires the acquisition of a series 
of social-behavioral competencies including the ability to self- 
regulate, initiate positive interactions with teachers and peers, 
attend to instruction, and engage in academic tasks (Walker, 
Ramsey, & Gresham, 2004). Children who are unsuccessful in meet- 
ing these expectations often experience teacher and peer rejection 
and have less than satisfactory teacher and peer relationships 
(Kegan, 1990). Unfortunately, there has been a sharp increase in the 
incidence of children who begin their school careers unable to navi- 
gate these expectations (McCabe, Hernandez, Lara, & Brooks-Gunn, 
2000). Children, whose serious school adjustment and behav- 
ior problems persist, are at risk for school social and emotional 
failure and detrimental outcomes later in life including possible 
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affiliation with disruptive peer groups, juvenile delinquency, tru- 
ancy, and school dropout (Patterson, Reid, & Dishion, 1992; Reid, 
1993). 

Intervening early in the school careers of these children is 
important and has been the focus of immense effort on the part 
of public, private, and national systems of education and research. 
Since the introduction of the Response to Intervention framework 
(Batsche et al., 2005), these efforts have been categorized based on 
a child’s educational and social needs at three levels: (a) univer- 
sal support (primary prevention), (b) targeted support (secondary 
prevention) and (c) intensive, individualized support (tertiary pre- 
vention). This approach, with its origin in the public health field, 
emerged as a model to address health concerns and evolved in 
the direction of public school application and subsequently early 
education. In a comprehensive review and analysis of more than 
2000 articles published between 1990 and 2006 on school-based, 
mental health interventions for at-risk students, Hoagwood et al. 
(2007) identified 64 methodologically rigorous studies for inclu- 
sion. Of these, 24 examined both educational and mental health 
outcomes, and only 1 5 of these studies showed a positive impact on 
both outcomes. Of the remaining 15,11 included home and school 
components with a focus on engaging and coordinating the efforts 
of parents and teachers. Hoagwood and her associates also noted 





2 


A.J. Frey et al. / Early Childhood Research Quarterly 30 (2015) 1-11 


that school interventions judged as effective for students requir- 
ing tertiary-level prevention strategies contain a well-designed 
and intensive family component to deliver the necessary strength 
and dosage levels to impact substantively school outcomes as well 
as address the focus on student’s social, emotional, and mental 
health problems. In addition to including a family component, 
the empirical literature advocates for interventions that are suffi- 
ciently flexible to be responsive to the unique situations and needs 
of families requiring intensive, individualized support. Hoagwood 
et al.’s review included effective secondary prevention programs 
such as The Incredible Years (Reid, Webster-Stratton, & Hammond, 
2003; Webster-Stratton, Reid, & Hammond, 2004) and First Step 
to Success (Walker et al., 1997), but did not include any programs 
designed to address the complex needs of children and families 
requiring tertiary level support. 

First Step to Success is an early intervention program designed 
for at-risk elementary school children in the primary grades (K-3) 
who show clear signs of emerging externalizing behavior patterns 
including aggression toward others, oppositional-defiant behavior, 
tantrums, rule infractions, and confrontations with peers and adults 
(Walker et al., 1997). The behaviorally at-risk student is the pri- 
mary focus of the First Step to Success program; however, teachers, 
peers, and parents are crucial intervention agents whose participa- 
tion is under the direction and supervision of a trained First Step 
behavioral coach. This professional is frequently a related service 
provider (e.g., school social worker, school counselor, school psy- 
chologist, behavioral specialists, special educator), and has overall 
responsibility for coordinating the intervention. 

The First Step intervention was developed through a model 
development grant ( 1 992-1 996) from the U.S. Office of Special Edu- 
cation Programs and was a cooperative effort between investigators 
at the University of Oregon, the Oregon Social Learning Center, 
and the Oregon Research Institute. In the past two decades, the 
First Step program has been the focus of a large number of federal 
and state-funded grants to support a range of research activities 
centering on its initial validation, replication, efficacy, and effec- 
tiveness. These grants have also supported examining the use of 
the program with students exhibiting elevated ADHD symptoma- 
tology (Seeley et al., 2009) and other student subpopulations (Feil 
et al., 2014; Frey, Small, et al., 2013). A recently released overview 
of the evidence base for the First Step to Success Early Intervention 
program summarizes research efforts and empirical outcomes that 
document the program as both efficacious and effective (Walker 
et al., 2014). The efficacy of the First Step intervention has been 
replicated repeatedly (Loman, Rodriguez, & Horner, 2010; Walker 
et al., 1998; Walker et al., 2009). Overall, this body of empirical 
evidence demonstrated the First Step intervention is socially valid, 
can be implemented with fidelity, and is associated with decreases 
in problem behavior, increases in social competency, and improve- 
ments in academically engaged time. A description of the First Step 
program’s complete research and development history along with 
its evidence-base is contained in Walker et al. (2012). This com- 
prehensive description also has appendices containing respectively 

(1) a listing of key First Step journal and chapter publications and 

(2) compilations of recommended lists of early interventions for 
behaviorally at-risk children (in which First Step was included) that 
were assembled and broadly disseminated by federal agencies and 
advocacy groups. 

A mixture of experimental, quasi-experimental, and replica- 
tion designs, involving group randomized and single case research 
methods, have been used to establish the First Step evidence 
base. First Step has been the focus of three randomized con- 
trolled trials to date— two of which were efficacy trials and one 
that was a national effectiveness study of the program’s effects 
involving five sites across the U.S. and 286 participants in grades 
I<-3. The First Step program has been implemented successfully 


in Canada, Australia, the Netherlands, and Turkey. First Step has 
also been successfully implemented with American Indian, African 
American, and Native Hawaiian students. In 2013, the First Step 
program was certified as a promising practice after a review 
by the What Works Clearinghouse of the Institute for Education 
Sciences. 

Walker et al. (2014) noted that students having the most severe 
impairments have highly variable and sometimes unsatisfactory 
responses to the First Step program. Additionally, this review 
demonstrates that the intervention consistently has less dramatic 
impact on behavior in the home than the school setting. One pos- 
sible explanation for the finding of inconsistent results with more 
severe children is that the homeBase component of First Step does 
not provide a similar intensity or dosage of the First Step interven- 
tion in the setting, as the school component does for the student in 
the classroom. Another explanation may be that the family com- 
ponent has not been successful at engaging and fostering parental 
motivation to change their parenting practices so as to positively 
impact child outcomes. 

In a classic study of parent noncompliance within mental health 
settings, Patterson and Forgatch (1985) demonstrated that thera- 
pists’ efforts to change parental behavior through direct teaching 
elicited immediate parent noncompliance, whereas efforts to sup- 
port parents decreased the likelihood of their noncompliance. 
Patterson and Chamberlain (1994) have systematically studied 
parental resistance, and concluded that parental motivation to 
change is a critical yet often neglected ingredient in improving 
parenting practices. Thus, the need for school mental health inter- 
ventions that include a home component and attend carefully to 
parent engagement, motivation, and follow through is substantial. 
In fact, the importance of engaging families is recognized as one of 
eight themes requiring systematic attention in order for the field of 
school mental health to advance (Weist, Lever, Bradshaw, & Owens, 
2014). 

Over the past four years, developers of the First Step inter- 
vention have been engaged in an iterative development process 
to create enhancements to the program that extend the range of 
the intervention. Two manualized enhancements of the First Step 
intervention were developed through this process. The first, the 
Tertiary First Step Resource Manual, is described in the methods 
section (Frey, Walker, et al., 2013c). The second, the First Step Class- 
room Check-up Resource Manual (Frey, Walker, et al., 2013d) can 
be implemented flexibly at the secondary and tertiary program lev- 
els, as a stand-alone intervention, or as one of several components 
of a yet-to-be-developed universal program variation within an 
overarching First Step System of Support. 

This manuscript reports an initial empirical study of the First 
Step program’s tertiary-level adaptation for more severely involved 
students. This adapted program variation differs from the orig- 
inal First Step in that it is designed for tertiary level students 
and includes (a) a new, more intensive home component (Ter- 
tiary homeBase), (b) screening procedures that require behavioral 
impairment in both home and school settings, and (c) modifications 
to the school component necessary for successful implementation 
with tertiary-level students. 

The purpose of this article is to report the feasibility and 
potential impact of the Tertiary First Step intervention. Specif- 
ically, we examined a number of process variables associated 
with these program enhancements, such as fidelity of imple- 
mentation, dosage, and satisfaction. Further, we examined the 
extent to which participation in the intervention was associated 
with reductions in parental distress and improvements in parent- 
ing efficacy, children’s social competency, and academic engaged 
time. Finally, we examined the associations between our pro- 
cess and outcome variables for the school and home components, 
respectively. 
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Method 

Participants 

Thirty-three families from Kentucky and Indiana participated 
in the classroom and home components of the Tertiary First Step 
feasibility study and 22 additional families completed baseline 
and post-intervention measures as part of a quasi-experimental, 
comparison group. Children participating in Tertiary First Step 
were from K-3 classrooms, ranged in age from 5 to 9 years old 
(M = 6.8 years, SD = 1.3), and were predominantly male (79%). The 
majority of students receiving Tertiary First Step qualified for 
free or reduced lunch (75%) and one third of participating chil- 
dren received special education services. Roughly one-third of 
participating children lived in two-parent households when they 
completed baseline data, according to parent self report. Prior to 
recruitment, the university-affiliated and school-affiliated institu- 
tional review boards approved the study. 

Intervention 

First Step to Success. First Step consists of three modular 
components designed to work together (though each can be 
implemented as a stand-alone procedure). These are (1) universal 
screening and identification, (2) school component, and (3) home 
component. 

The school component of First Step was developed by Hops and 
Walker (1988). The program begins with a 20-minute daily imple- 
mentation period that is gradually extended to the entire school 
day. Initially, the coach, in close proximity to the target child, moni- 
tors her or his classroom behavior using the green or red card. The 
coach awards points if the child’s behavior is appropriate, based on 
an interval system. When the daily performance criterion of 80% 
or more of the available points is met, a brief rewarding activity 
involving the target child and peers is made available immedi- 
ately and a daily home note also communicates the results of the 
game to the parents. Parents then provide positive reinforcement 
with an individual activity or reward immediately when the child 
returns home. If the criterion isn’t met, that program day is then 
repeated and/or the child is recycled to an earlier, successfully com- 
pleted program day before proceeding. The school component of 
First Step requires 30 program days for successful completion. The 
coach phase (program days 1 -1 0) is the responsibility of a First Step 
behavioral coach, who coordinates the implementation process. 
The teacher phase (program days 10-20) is operated by the teacher 
who assumes control of the program’s operation on program day six 
but with close supervision and support from the coach. It is during 
the teacher phase that the program is gradually extended to include 
the entire school day. The maintenance phase lasts from program 
day 21 to 30 after which the formal school intervention ends. In 
this phase, the target child is rewarded primarily with praise and 
expressions of approval or recognition from the teacher and peers 
at school and the parents at home. An attempt is made during this 
phase to reduce the child’s dependence on the program by substi- 
tuting adult recognition for points, reducing the amount of daily 
feedback given, and making occasional rewards contingent upon 
exemplary performance. 

After the transition to the teacher phase, the coach’s focus turns 
from school to home. The coach invites the focus student’s par- 
ents to participate in homeBase, which involves learning how to 
teach school success skills at home and to collaborate with the 
teacher and coach to facilitate the transfer and demonstration of 
these skills at school. Over a six-to-eight-week period, parents 
meet weekly with the First Step coach, usually in their home, 
and participate in homeBase via reading, discussion, role-play, and 
demonstrations. Each week’s meeting focuses on one skill with 


review and discussion of previously learned skills as needed. The 
specific homeBase skills taught are: communication and sharing, 
cooperation, limit setting, problem solving, friendship making, and 
self-confidence. Parents are provided with a manual containing all 
the information, guidelines, and accompanying materials needed to 
implement homeBase. These materials provide a useful reference 
for parents and the coach during and following implementation of 
the First Step program. The coach provides support, supervision, 
and trouble shooting of any problems and issues that arise during 
and following the program’s implementation, while serving as a 
communication bridge between the parent and school. 

Tertiary First Step. Michie, van Stralen, and West (2011) have 
developed a framework for intervention designers that places a 
premium on participant motivation. These authors note that many 
intervention designers develop new interventions without relying 
on a framework to guide and rationalize the creation of various 
components, in part because useful frameworks do not exist. In 
response to this need, they created the Behavior Change Wheel to 
provide a basis for designing interventions. The behavior system 
in this framework is comprised of three components that interact 
dramatically to influence behavior change at the individual level: 
capability, opportunity, and motivation. Capability, which includes 
knowledge and skills, involves the individual’s psychological and 
physiological capacity to alter the behavior change target. Oppor- 
tunity is comprised of factors that are external to the individual 
that prompt behavior or make it possible. Motivation is defined as 
all of the “brain processes that energize and direct behavior” (p. 4). 
Michie et al. suggest that a given intervention might change only 
one or more of these components, and that an initial task of inter- 
vention developers is to consider what the behavior target of the 
intervention is, and what components of the system need to be 
addressed to achieve the desired goals. 

The importance of parental motivation has led to an increase in 
collaborative approaches for caregivers of students with challeng- 
ing behavior (Frey et al., 2011; Smith, Dishion, Shaw, & Wilson, 
2013; Smith, Handler, & Nash, 2010; Smith, Wolf, Handler, & 
Nash, 2009). Recent efforts in this context have adopted strate- 
gies from motivational interviewing. Motivational interviewing 
is a burgeoning approach to more effectively influence parents’ 
engagement and behavior change. Miller and Rollnick (201 2 ) define 
motivational interviewing as “a collaborative, goal-oriented style 
of communication with particular attention to the language of 
change” and go on to say “. . .it is designed to strengthen per- 
sonal motivation for and commitment to a specific goal by eliciting 
and exploring the person’s own reasons for change within an 
atmosphere of acceptance and compassion” (p. 29). Motivational 
interviewing is based on the belief that how one talks about change 
is related to how they act. Simply stated, the more one talks about or 
argues for change, the more likely it is he or she will change. Con- 
versely, the more one verbalizes reasons against change, the less 
likely he or she is to change. Motivational interviewing, therefore, 
is an approach that helps accelerate the change process “by liter- 
ally talking oneself into change" (p. 168). Developing a supportive 
environment/relationship and evoking change talk, or any self- 
expressed language that is an argument for change is critical in the 
facilitation of motivational interviewing. The evidence for motiva- 
tional interviewing provides compelling verification for the notion 
that the therapist can influence clients’ expression of change talk 
and that there is a relationship between change talk and behavior 
(Forgatch & Patterson, 1985; Glynn & Moyers, 2010; Miller, Yahne, 
Moyers, Martinez, & Pirritano, 2004; Moyers & Martin, 2006). 

Nock and Kazdin (2005) pioneered the application of moti- 
vational interviewing in the context of parenting with their 
Parent Enhancement Intervention, a model that assesses caregiver 
perception of readiness and that attempts to improve parental 
engagement and adherence (i.e., attendance). Additionally, The 
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Family Check-up, which is part of the multi-component EcoFIT 
intervention, includes three brief, family-centered sessions to moti- 
vate caregivers to change parenting practices and use intervention 
services addressing their specific needs (Dishion & Stormshak, 
2007). 

Tertiary First Step includes three components of First Step 
that have been enhanced for children with extremely challeng- 
ing behavior who require tertiary-level prevention strategies. The 
first includes screening procedures for identifying appropriate 
participants. The second is very similar to the original school 
component, but includes modifications often necessary for suc- 
cessful implementation with tertiary-level students. Examples of 
school component modifications include: (1 ) completion of a func- 
tional behavioral assessment; (2) increased coach-teacher contact; 
(3) participation in current service coordination efforts internally 
within the school and with external service providers; and (4) 
reductions to the length of the First Step program (in total num- 
ber of days). Finally, Tertiary homeBase, the home component, 
is an adaptation of motivational interviewing. Since the Tertiary 
homeBase component represents the primary enhancement to the 
tertiary First Step variation, it alone is described below. This study 
presents the first empirical examination of the Tertiary Frist Step 
variation. 

Tertiary homeBase. Tertiary homeBase addresses all three com- 
ponents of Michie et al.’s (2011) Behavior Change Wheel. Capability 
is addressed by including content consistent with the theoretical 
and empirical literature concerning family management prac- 
tices and the development of children with challenging behavior 
described in the introduction. During the Tertiary homeBase inter- 
vention, parents are encouraged to modify their parenting practices 
consistent with one or more of the five universal principles of posi- 
tive behavior support that are central to the First Step intervention: 
(1 ) establish clear expectations; (2) directly teach the expectations; 
(3) reinforce the display of expectations; (4) minimize attention 
for minor inappropriate behaviors; and (5) establish clear conse- 
quences for unacceptable behavior (Sprague & Golly, 2013). 

Addressing the motivation component of the Behavior Change 
Wheel was a primary focus of our Tertiary homeBase development 
efforts. Because no existing processes, models, or frameworks, 
based on the motivational interviewing approach, were avail- 
able to inform a detailed procedural write up for its application 
across home and school settings, our efforts to adapt motiva- 
tional interviewing for this purpose resulted in the development of 
the Motivational Interviewing Navigation Guide. The Motivational 
Interviewing Navigation Guide is a process for increasing intrin- 
sic motivation to adopt and implement an evidence-based practice 
with integrity, applicable to both school and home settings. The 
five steps include: (1) engage in values discovery; (2) assess cur- 
rent practices; (3) share performance feedback; (4) offer extended 
consultation, education & support; and (5) provide closure (Frey, 
Lee, et al., 2013a; Frey, Lee, et al., 2013b; Lee et al., in press). It is 
this conceptualization that was infused into the Tertiary homeBase 
intervention procedures to increase parents’ motivation to change 
their behavior. 

The motivational component is addressed at each of the Moti- 
vational Interviewing Navigation Guide steps; it is attended to 
constantly during steps 1 and 2, and as needed in steps 3 and 
4. The capability of the caregiver, primarily parenting knowledge 
and skills, is addressed in steps 3 and 4. Opportunity is addressed 
throughout the intervention since it occurs in the home, and specif- 
ically in step 4 with creation of a behavior change plan which 
individualizes the application of new knowledge and skills to 
address routines or family tasks that parents have identified as 
important, needing attention, and consistent with their values. 
Alternatively, parents can choose to complete any of the six orig- 
inal homeBase skill-focused curricular topics. A summary of the 


intervention activities is provided in Table 1. Tertiary homeBase 
typically includes two to five home visits designed to increase their 
motivation and capacity to implement effective parenting prac- 
tices. The Tertiary First Step to Success Resource Manual provides 
detailed support for interventionists (Frey, Walker et al., 2013c). 

Recruitment and screening. Project staff recruited teachers 
across two cohorts to participate in a feasibility study of the Ter- 
tiary First Step to Success intervention. We used a two-step process 
incorporating teacher and parent report to identify students eligi- 
ble for inclusion in the study. At step 1, teachers completed the 
first two stages of the Systematic Screening for Behavior Disorders 
(SSBD; Walker & Severson, 1990). At stage 1, teachers identified 
five students within their classrooms who were at elevated risk for 
externalizing behavior problems. At stage 2, teachers completed 
brief behavior rating scales for each of the students identified at 
stage 1 . Stage 2 data were used to (a) identify the students who met 
SSBD criteria, (b) rank order students within classrooms in terms 
of severity, and (c) target the highest ranked student in each class- 
room. At step two, we collected the externalizing scale of the Child 
Behavior Checklist (CBCL; Achenbach, 1991) from the parents of the 
highest ranked student to verify the child’s behavioral status across 
school and home settings. If the student met criteria on the parent- 
reported CBCL (T Score > 60), we recruited the family to participate 
in the study. Thus, for each classroom, the highest ranked student 
who met SSBD screening criteria and CBCL screening criteria were 
eligible to participate in the study. If the highest-ranked student on 
the SSBD did not meet CBCL criteria, we repeated the process with 
the next highest rank student in the classroom. The first step of the 
screening process (SSBD) is consistent with the screening proce- 
dures for the original First Step intervention. However, requiring 
parents to document substantial impairment in the home setting 
is unique to the tertiary application. We provided teachers $20 to 
complete the screening process. 

We recruited teachers for the Tertiary First Step intervention 
across two cohorts during the 2010-2011 and 2011-2012 school 
years. Participating teachers were from ten elementary schools in 
Kentucky and Indiana. Seventy of 78 consented K-3 teachers (90%) 
participated in SSBD screening, completing stages 1 and 2 for 268 
students. Of the 70 teachers completing screening, 33 (47%) had 
an eligible consented student who participated in the study and 
received the intervention. For the remaining teachers, we were 
unable to identify (n = 14) or obtain consent for a student who met 
full inclusion criteria (n = 23). 

Research staff recruited teachers for a quasi-experimental com- 
parison group during the 2011-2012 school year to examine 
between-subject effects and control for potential history effects. 
Comparison-group teachers were not recruited from the same 
schools as the experimental-group teachers. We utilized the same 
screening and inclusion criteria for the comparison classrooms. 
Thirty teachers completed SSBD gates 1 and 2 for 149 students. 
Twenty-six of the 30 classrooms (86.7%) had at least one student 
who met SSBD eligibility criteria. Twenty-two students met full 
inclusion criteria (i.e., SSBD and CBCL criteria described above). 
For the remaining eight classrooms, students did not meet inclu- 
sion criteria (n = 4) or project staff were unable to recruit the 
family to participate (n = 4). Students and families in the quasi- 
experimental comparison group may have been receiving school- 
or community-based intervention and support services, but did not 
receive intervention support from our research team prior to the 
collection of posttest data. They were offered a home-based con- 
sultation following the collection of posttest data. 

Training and support. Participating coaches were employees of 
the University of Louisville. All three coaches had Masters Degrees: 
one in education, one in social work, and one in school counseling. 
Prior to the study, the three coaches had no previous experience 
with motivational interviewing. All three received three days of 
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Table 1 

Summary of the Tertiary homeBase intervention activities. 
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MING Step # and title (duration) 


Description 


The main purpose of this step of the MING is to learn about the family through an 
ecological assessment, and discover the parent’s values, goals, and hopes for their 
children’s future using a values discovery activity. 

Successfully completed the universal principles interview and observational protocol 
associated with this step of the MING allow the coach to learn about existing parenting 
practices that are consistent with - or potentially in conflict with - the universal principles. 
This information is used to structure the provision of performance feedback in Step 3. 

Step 3 of the MING is used to provide performance feedback. The debriefing interview is 
structured to encourage the parent to reflect on their implementation of the universal 
principles, and if necessary, increase the extent to which they believe implementing the 
principles is important. At the end of the interview, parents are given the option of ending 
the consultation relationship or replicating Steps 2 and 3 after having articulated specific 
goals for improvement. 

Step 4 Extended Consultation, Education, and Support (60 minutes) During this (optional) step of the process, the coach and parent negotiate the specifics of a 

behavior change plan. Once the specifics are decided upon and step 4 is completed, the 
coach may deem it appropriate to take an educational stance, more freely offering advice 
and teaching skills through consultation, or direct education. We recommend educational 
strategies including modeling, role-playing, pre-correcting for implementation problems 
by exploring barriers to implementation, and the more Ml focused strategy of 
Elicit-Provide-Elicit (EPE). If the parent chooses to participate in this aspect of the 
intervention. Steps 2 & 3 are repeated in order to provide continuing support for 
parent-established goals. 

Step 5 Closure (60 minutes) Whether a parent selects closure due to high confidence in their ability to change on their 

own or due to low motivation, steps should be taken to insure that the relationship ends on 
a positive note and that the parent leaves with tools they may choose to use in the future. 


Step 1 Engage in Values Discovery (60 minutes) 


Step 2 Assess Current Practices (60 minutes) 


Step 3 Share Performance Feedback (60 minutes) 


pre-intervention training and a year of ongoing professional devel- 
opment support learning how to apply motivational interviewing 
skills within the context of our intervention protocols. Their pre- 
intervention training was provided by a substance abuse clinician 
with expertise in motivational interviewing and professional devel- 
opment involved weekly supervision sessions over the first year of 
implementation. As part of their professional development, they 
were required to obtain audio recordings of their interactions 
with teachers and parents. These recordings were analyzed and 
processed in the context of group supervision. Frey, Lee, et al. 
(2013a) provides a detailed description of the training procedures, 
as well as coaches’ proficiency applying motivational interviewing 
with parents. 

Teachers participating in the Tertiary First Step intervention 
received a 3-hour training describing the intervention procedures. 
Coaches assisted the teachers with the school component imple- 
mentation. 

Data collection. Prior to beginning the intervention, parents 
and teachers participating in Tertiary First Step completed a base- 
line questionnaire containing demographic and outcome measures. 
At post-intervention, participants completed two questionnaires: 
one containing outcome measures and another containing pro- 
cess measures addressing program satisfaction and barriers to 
participation. Teachers and parents from the comparison group 
completed an outcome questionnaire at each time point but did not 
complete process measures given that they didn't receive the inter- 
vention. For the comparison group, baseline and post measures 
were collected roughly 60 days apart to approximate the window of 
time between baseline and post-intervention for the intervention 
group. Teachers and parents from the intervention group and com- 
parison group received $50 for completing a questionnaire at each 
time point (i.e., $ 1 00 for completing baseline and post-intervention 
packets). For participants receiving the intervention, coaches dis- 
tributed questionnaires to parents and teachers. For participants 
in the comparison group, project staff distributed questionnaires. 
Participants returned completed questionnaires via mail using a 
postage-paid envelope distributed with the questionnaire or, when 
needed, project staff or the coach collected the questionnaire from 


the participant. Direct observation data (described below) were 
collected at baseline and post-intervention for all students par- 
ticipating in Tertiary First Step but were not collected for the 
comparison group due to time and budgetary constraints. 

Fidelity 

A 20-item implementation fidelity checklist was used to eval- 
uate the fidelity of implementation for the school component. 
This observer-completed measure assesses the extent to which the 
coach and teacher adhere to implementation guidelines for the 
school component of the First Step program. For each question, the 
observer indicates (a) whether the component was implemented 
and (b) the quality of implementation. Adherence items are scored 
dichotomously (i.e„ yes or no) and quality items are scored on 
a 5-point scale (0 = very poor, .25= poor, .50 = okay, .75= good, to 
1.0 = excellent). Adherence scores were calculated as the propor- 
tion of procedures correctly implemented. An observer collected 
the fidelity data on three occasions: once during the coach phase 
and twice during the teacher phase. Adherence to 80% or more of 
observed program components represents adequate adherence and 
quality ratings of .75-.90 represent adequate levels of implementa- 
tion quality. We used the data from the fidelity checklist to compute 
adherence and implementation quality scores. Measures of teacher 
and coach implementation quality represent the mean quality rat- 
ing across the observed program components. We also calculated 
overall adherence and quality measures (i.e., the mean of the two 
implementers) across both the coach and teacher. 

Dosage 

The monitoring log, completed by the coach and teacher, is 
used to record the focus student’s daily participation in the school 
component. Upon completion of the intervention, it provides a 
summary of the total number of program days completed, the num- 
ber of program recycle days, and a summary of the points and 
rewards earned daily by the child. In accordance with other studies 
of the First Step program (Sumi et al„ 2013; Walker et al., 2009), 
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we calculated classroom dosage as the proportion of program days 
delivered out of the possible 30 available days. 

A checklist was also completed by the coach to measure the 
dosage for the home component. This checklist required coaches 
to document the number of the steps of the Tertiary homeBase 
intervention completed. Dosage was calculated as a percentage of 
steps completed. 

Social validity 

Social validity items for all informants were scored on a five- 
point Likert scale from strongly disagree to strongly agree. Parent 
report included 12 items that assess usability, support, and effec- 
tiveness of the program in the home setting. Coefficient alpha (a), 
a measure of internal consistency reliability, is .93 for the 12-item, 
parent-reported satisfaction scale. Subsequent coefficient alphas 
reported below for social validity and outcome measures are esti- 
mated from sample data collected for this study. Teacher report 
included a 1 3-item scale assessing satisfaction with the school com- 
ponent (a = .93). The coach also completed a 6-item satisfaction 
scale (a = .84) pertaining to the compatibility and effectiveness of 
the classroom component, and six items (a = .90) addressing the 
compatibility and effectiveness of the home components of the pro- 
gram (i.e., Tertiary homeBase). For each measure, we calculated a 
mean rating across items. 

Outcomes 

Parent outcomes. Parental distress and parenting efficacy 
served as proximal outcomes, and were measured at baseline and 
post-intervention. The parental distress subscale of the Parenting 
Stress Index - Short Form (PSI-SF; Abidin, 1995) determines the 
distress a parent is experiencing in his or her role as a parent 
and expresses the distress as a function of personal factors that 
are directly related to parenting. The subscale includes 12 items 
(a = .92) scored on a 5-point rating scale. Total raw scores range 
from 12 to 60 with higher scores indicating increased levels of 
parental distress. 

The Parenting Ladder (NCP Research, 2000) measures parent- 
reported self-efficacy for parenting. The Parenting Ladder consists 
of 6-items (a = .91). Parents rate confidence in their parenting 
knowledge and abilities on a 7-point rating scale ranging from low 
to high. Total raw scores range from 0 to 42 with higher scores 
indicating higher levels of perceived self-efficacy. 

Teacher- and parent-reported outcome measures. The Social 
Skills Improvement System Rating Scales (SSiS; Gresham & Elliott, 
2008) is a multi-informant assessment tool that measures (a) 
social behaviors that facilitate positive interactions with peers, tea- 
chers, siblings, and parents, (b) problem behaviors that impede 
the acquisition of social skills, and (c) general academic function- 
ing. The teacher-completed version of the measure includes 46 
social skills items (a = .90), 30 problem behavior items (a = .74), 
and 7 academic competence items (a = .96). The parent-completed 
version includes 46 social skills items (a = .93), and 33 problem 
behavior items (cr = .82). Items are rated on a 4-point frequency 
scale (Never, Seldom, Often, and Almost Always). Academic com- 
petence items, scored on a 5-point scale from lowest 10 % to 
middle 40% to highest 10%, assess the student’s reading and math 
skills, motivation, cognitive functioning, and parental support 
relative to his or her classmates. We converted raw scores to 
standard scores using gender-specific normative data from the SSiS 
manual. 

The SSBD Combined Frequency Index was collected at screening, 
baseline, and post-intervention. This index, part of the SSBD stage- 
2 rating scales (Walker & Severson, 1990), includes the Adaptive 
Behavior Index (ABI) and Maladaptive Behavior Index (MBI). The 


ABI and MBI are 12-item (a = .88) and 11-item (a = .86) scales, 
respectively, that assess a student’s adaptive and maladaptive 
behavioral adjustments with interactions with teachers and peers. 
Items are scored on a 5-point rating scale ranging from never to fre- 
quently. The SSBD is nationally normed, has excellent psychometric 
properties, and has been used in a number of research studies 
(Seeley et al., 2009; Severson, Walker, Hope-Doolittle, Kratochwill, 
& Gresham, 2007; Walker et al., 2009). Raw scale scores were com- 
puted for each measure with higher scores on the ABI indicating 
better levels of adaptive functioning and higher scores on the MBI 
indicating higher levels of maladaptive functioning. 

Observation outcome measures. Project staff collected Aca- 
demic Engaged Time (AET) data using a direct-observation measure 
(three 20-minute observations), on separate days at baseline and 
again at post-intervention (Walker & Severson, 1990). For each 
time point, we computed the mean percent of AET across the three 
observations. AET is an estimate of the amount of time a stu- 
dent spends engaged in academic activities and is an important 
indicator of a student’s academic success and adjustment to class- 
room expectations. We collected inter-rater reliability for 15% of 
collected AET observations at baseline and at post intervention. 
The intra-class correlation assessing inter-rater reliability for the 
AET was excellent at baseline (ICC[3,1 ] = .98) and post-intervention 
(ICC[3,1 ] = .94). 

The Peer Social Behavior (PSB) coding system was also recorded. 
The PSB is a partial-interval observation procedure used to record 
the percentage of intervals the target student is engaged in pos- 
itive and negative interactions with peers in unstructured or 
semi-structured settings (Walker & Severson, 1990). Project staff 
conducted three 20-minute observations at baseline and three at 
post-intervention on separate days using an adapted version of 
the instrument. Over the 20-minute session, observers recorded 
at one-minute intervals whether the student was engaged in pos- 
itive social engagement, negative social engagement, parallel play, 
or was playing alone. For each time point, we aggregated data 
from the three observation sessions and calculated the percent 
of positive and negative engagement by dividing the number 
of positive engagement intervals and the number of negative 
engagement intervals by the total number of intervals observed. 
We collected inter-rater reliability data for 15% of baseline and 
post-intervention observations. The intra-class correlation for pos- 
itive interactions was excellent at baseline (ICC[3,1 ] = .97) and 
acceptable at post-intervention (ICC[3,1 ] = .86). For negative inter- 
actions, the intra-class correlation was also acceptable at baseline 
(ICC[3,1] = .89) and post-intervention (ICC[3,1] = .91). 

Statistical analyses 

We examined between-subject and within-subject effects on 
teacher and parent-reported outcome measures. To evaluate 
between-subject effects, we estimated a series of covariate- 
adjusted regression models using Mplus 6.0 statistical software 
(Muthen & Muthen, 1998-2010). For the regression models, each 
outcome was regressed on a dichotomous variable indicating 
intervention group (1 =EFS group, 0 = comparison group) and one 
covariate, the baseline value of the outcome. Preliminary mod- 
els included an interaction term (i.e., intervention group x baseline 
value of the outcome) to test that the slopes of the regression lines 
were equivalent for each group. If non-significant, the interaction 
term was removed from the model. 

For the comparison group, only parent- and teacher-reported 
outcomes were collected. In turn, for the intervention group, we 
also examined within-subject effects for our observation measures 
and primary teacher- and parent-reported outcomes. We examined 
within-subject effects in an analysis of variance (ANOVA) frame- 
work using the general linear model (GLM) procedure in SPSS 19. 
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For the between-subject analysis, we report Hedges’ g as a 
measure of effect size. The What Works Clearinghouse (WWC) 
recommends Hedges’ g as the preferred measure of effect size for 
continuous outcomes. Hedges’ g, the standardized mean difference, 
is calculated by taking the difference between the mean outcome 
of each group and dividing it by the pooled within-group standard 
deviation (WWC, 2011). Effect sizes of .2 are considered small, .5 
are considered medium and .8 are considered large effects. For the 
within-subject analysis, we report partial point-biserial r as a mea- 
sure of effect size (Rosnow & Rosenthal, 2008). Effect sizes of .14, 
.36, and .51 are considered small, medium, and large, respectively, 
for the partial r (Cohen, 1988). 

We also report the WWC (201 1 ) improvement index as a mea- 
sure of practical significance. To calculate the improvement index, 
we (a) converted each effect size estimate to a Cohen’s U3 index 
using a standard normal distribution z-score table and (b) sub- 
tracted the U3 index from 50%, the percentile rank of an average 
student in the comparison group. The WWC improvement index 
represents the expected change in percentile rank for an average 
student in the comparison sample if that student had received the 
Tertiary First Step intervention. 

Finally, to examine associations between these process meas- 
ures and change in parent and child outcomes, we specified 
covariate-adjusted regression models. We regressed post interven- 
tion outcomes on the baseline value of the outcome measure and 
the process measure of interest. We anticipated that process meas- 
ures specific to the classroom component of the program would 
have the strongest associations with school-based or teacher- 
reported student outcomes and that process measures specific to 
the home component would have the strongest associations with 
parent outcomes and parent-reported student outcomes. Thus, we 
examined whether dose, adherence, and implementation quality 
of the school component were associated with change in teacher- 
reported prosocial behavior (i.e„ ABI and SSiS social skills), problem 
behavior (i.e., MB1 and SSiS problem behavior), and academic com- 
petence (i.e., SSiS academic competence, and AET). For the home 
component, we examined whether parent-reported satisfaction, 
coach-reported parent compliance, and dose were associated with 
improvements in parenting efficacy, reductions in parental distress, 
and changes in parent-reported prosocial and problem behavior as 
reported on the SSiS. 

Missing data 

We used the full information maximum likelihood (FIML) 
estimator in MPlus 6.0 to address missing data in the covariate- 
adjusted regression models. FIML, a state-of-the-art technique for 
handling missing data, uses all available data to calculate unbiased 
parameter estimates and standard errors (Schafer & Graham, 2002). 
We included seven auxiliary variables in the models (i.e., child’s 
SSBD rank, child’s gender, parent's current marital status, parent’s 
education level, estimated annual household income, number of 
children in the parent’s household, and teacher-reported num- 
ber of years teaching) as potential correlates of missingness in 
order to improve the accuracy of FIML estimation. Potential cor- 
relates of missingness increase statistical power, reduce bias, and 
improve the plausibility of the missing at random assumption with- 
out altering the interpretation of parameter estimates (Collins, 
Schafer, & Kam, 2001; Enders, 2010). 

Results 

Fidelity 

During coach and teacher phases of the school component, 
Tertiary First Step program adherence was excellent. Coaches 


implemented 96% (range = 64-100%) and teachers implemented 
90% (range = 63 -100%) of the school component. Implementation 
quality was excellent during the coach phase (.96; range = .90-1 .00) 
and good during the teacher phase (.84; range = .61-1.00). 

Dosage 

Students received, on average, 78% (SD = 28%) of the requi- 
site program days for the school component. Twenty-five parents 
(76%) completed 75% or more of the steps of the home compo- 
nent as described in Table 1. Seven parents completed two or 
fewer steps and were considered non-completers. Of the 25 par- 
ents who completed the required components, 14(60%) committed 
to and developed a change plan, and most (87%) of those parents 
implemented it. Seven families were referred for community-based 
mental resources to receive on-going, additional support for the 
child or family, based on coach discretion. Eight parents (24%) 
ended their participation in the program before the formal invi- 
tation for closure was offered. 

Social validity 

Parents’ responses to the satisfaction questionnaire ranged from 
3.2 to 5.0 with mean scores of 4.6 (SD = 0.5) on the 12-item scale. 
Item-level means were above 4.0 on all items. The lowest mean 
rating was in response to a question asking whether the program 
had a positive effect on the rest of the family (AljSDj = 4.3[0.8]) and 
the highest mean rating was in response to a question about the 
goals of the program being clearly explained (M[SD] = 4.8[0.4] ). Our 
coaches reported satisfaction that can be classified as moderate 
overall (M = 3.73, SD = 0.74). 

Outcomes 

Baseline equivalence. Students who received the tertiary ver- 
sion of the First Step intervention did not differ significantly from 
the comparison group on baseline behavioral and academic out- 
come measures and most student, parent, and teacher demographic 
characteristics. Table 2 contains a summary of student demo- 
graphic and behavioral characteristics for each group. The two 
groups differed only on the number of African American students 
in the comparison sample as compared to the Tertiary First Step 
condition (68% vs. 39%, respectively). 

Although there were no statistically significant differences 
between the groups with respect to parent demographic character- 
istics, there were a disproportionate number of African American 
parents in the comparison group (61%) as compared to the interven- 
tion group (42%). Other parent demographic characteristics were 
comparable across the groups. Parents in the intervention condition 
had a mean age of 38 years (SD = 1 0.4), were primarily female (88%), 
and were predominantly the biological or adoptive mother of the 
participating student (81%). Nearly 30% reported having an Asso- 
ciate’s degree or higher and the majority were currently employed 
(61%). Approximately 36% of participating students lived in two- 
parent households. Parents in the comparison group had a mean 
age of 35 years (SD = 9.2), were predominantly female (96%), and 
were the biological or adoptive mother of the student (82%). Thirty- 
two percent had an Associate’s degree or higher and 68% were 
employed. Roughly 32% of students in the comparison group lived 
in a two-parent household. 

There were no differences on teacher characteristics between 
the two groups. All teachers participating in the Tertiary First Step 
intervention reported being the lead teacher of the classroom. The 
majority were female (93.3%) and half reported having a Master’s 
degree or higher. Teachers reported having worked in the field 
for an average of 14.1 years (SD = 8.8) and had taught students 
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Table 2 

Baseline equivalence of student demographic and behavioral characteristics. 



Total (n = 55) 

Comparison (n = 22) 

Tertiary FS (n = 33) 

Test statistic 

p-Value 

Demographic characteristic 






Age, M(SD) 

7.0 (1.2) 

73(1.1) 

6.8(13) 

1.53 

.133 

Percent female 

20.0 

18.2 

21.2 

0.08 

.783 

Percent African American 

50.9 

68.2 

39.4 

4.38 

.036 

Percent Caucasian 

36.4 

22.7 

45.5 

2.95 

.086 

Percent free/reduced lunch 

82.2 

90.5 

75.0 

1.84 

.176 

Percent IEP 

25.5 

13.6 

33.3 

2.70 

.100 

Screening measures 






SSBD stage 2 rank 




2.78 

.249 

Percent ranked 1 st 

69.1 

59.1 

75.8 



Percent ranked 2nd 

23.6 

27.3 

21.2 



Percent ranked 3rd 

7.3 

13.6 

3.0 



Percent in clinical range on CBCL externalizing scale 

88.9 

85.7 

90.9 

0.35 

.554 

Critical Events Index, M(SD) 

8.1 (3.2) 

8.5 (3.1) 

7.8 (3.2) 

0.90 

.370 

Adaptive Behavior Index, M(SD) 

29.6(6.3) 

30.5 (7.0) 

29.0(5.8) 

0.85 

.401 

Maladaptive Behavior Index, M(SD) 

38.2 (6.9) 

39.7(6.3) 

37.1 (7.1) 

1.39 

.170 


Notes: SSBD, Systematic Screening for Behavior Disorders; CBCL, Child Behavior Checklist. 


who receive special education services for an average of 1 1 years 
(SD = 9.1). 

Attrition and missing data. For the Tertiary First Step group, 
data were available for 94% of teachers and 94% of parents at base- 
line. At post-intervention, 32 teachers (97%) and 28 parents (88%) 
returned a questionnaire. For the comparison group, data were 
available for all 22 teachers and parents at baseline, all teachers 
at post intervention, and 19 parents (86%) at post intervention. 
Students with complete assessment data from the comparison 
group did not differ from those with a missing assessment. Ter- 
tiary First Step students with complete data across time points 
and informants did not differ significantly from those with missing 
assessments on student demographics, student behavioral char- 
acteristics, or parent characteristics. The two groups did differ, 
however, on the number of years the teacher had been working 
in the field. The teachers of students with complete data had been 
working in the field longer than the teachers of students with miss- 
ing data (15.6 years [SD = 9.0] as compared to 7.7 years [SD = 4.1], 
respectively). 

Between-subject results. Results from the covariate-adjusted 
regression models as well as baseline and post-test intervention 
means and standard deviations for the Tertiary First Step and com- 
parison conditions are presented in Table 3. For the three teacher- 
and parent-reported outcomes in the pro-social behavior domain, 
students who received the tertiary version of First Step had sta- 
tistically significant improvement in adaptive behavior and social 
skills at post-test as compared to students in the comparison 
sample. Hedges’ g effect sizes for the three pro-social outcomes 
ranged .36-1.11. Students who participated in the intervention also 
had statistically significant reductions in maladaptive and prob- 
lem behaviors across both school and home settings. The Hedges’ g 
effect sizes for the teacher- and parent-reported problem behaviors 
ranged from -.77 to -1.17. There were no statistically significant 
changes in student academic competence after completion of the 
intervention (Hedges’ g = . 19). 

Within-subject results. We also examined within-subject 
effects for our observation measures, parenting measures, and 
primary teacher- and parent-reported outcomes. Within-subject 
effect sizes for the academic domain were .25 for academic com- 
petence (F[ 1 ,29] = 1.80, p = .190) and .82 for AET (F[ 1,32] = 62.17, 
p < .001 ). After receiving the intervention, student AET improved 
on average from 59% (SD = 17%) to 75% (SD = 16%). Effects for the 
PSB observation data were in the medium to large range. The effect 
size for positive interactions (F[ 1 ,30] = 6.04, p = .020) was .42 and .53 
for negative interactions (F[ 1, 30] = 11.59, p = . 002). Positive inter- 
actions with peers increased from baseline (M[SD] = 27.3[16.5]) 


to post intervention (MjSD/ = 35.2[17.1]) and negative interactions 
with peers decreased from 4.3 (SD = 4.1 ) to 1.7 (SD = 1.9). 

For the parenting measures, mean scores on the parental dis- 
tress subscale decreased from 26.7 ( SD = 10.8) at baseline to 24.0 
(SD = 8.9) at post-test. The Ftest for the within-subjects ANOVA was 
not statistically significant (F[l,27] = 3.51, p = . 072) and the effect 
size was small (partial r=.34). Mean scores on parenting efficacy 
increased from 4.60 (SD = 1.10) at baseline to 5.00 (SD = 0.80) at 
posttest. The effect size for increases in parental efficacy was .41 
(medium), and the differences from baseline to post-test were sta- 
tistically significant (F[l,27] = 5.53, p = .026). 

For the parent- and teacher-reported outcomes, within-subject 
partial r effect sizes were .57, .66, and .74 for teacher-reported AB1, 
SSiS social skills, and parent-reported SSiS social skills, respectively, 
within the pro-social domain. For the problem behavior domain, 
effect sizes were .70 and .65 for teacher-reported MB1 and SSiS 
problem behavior, and .79 for parent-reported SSiS problem behav- 
ior. 

Practical significance. The mean improvement index score for 
outcomes in the pro-social behavior domain was +28 percentile 
points (i.e., if an average control student received the EFS interven- 
tion, we could anticipate a mean improvement of 28% on pro-social 
outcomes). The improvement index for teacher-reported adaptive 
behavior was +31 percentile points and +37 percentile points for 
social skills. The improvement index for parent-reported social 
skills (+14.8 percentile points) was more modest. For the problem 
behavior domain, mean improvement across the three outcomes 
was +33 percentile points. Teacher-reported problem behavior out- 
comes ranged from +34 to +38 percentile points for maladaptive 
and problem behavior, respectively. Parent-reported improvement 
in problem behavior scale was +28 percentile points. There were 
positive improvements across all primary outcomes and settings. 
Mean improvement in the home setting was +34 percentile points 
and mean improvement in the home setting was +21 percentile 
points. 

Process-outcome analysis. For the school outcomes, dose was 
associated with improvements in teacher-reported social skills 
(t = 2.60, p = .009) and reductions in teacher-reported problem 
behavior ( t = -3.49, p < .001 ). For a one standard deviation increase 
in dose, post-intervention, teacher-reported social skills increased 
by .31 standard deviations and teacher-reported problem behavior 
decreased by .40 standard deviations. There were no statistically 
significant associations between dose and the other outcome meas- 
ures. As well, there were no statistically-significant associations 
between school-based or teacher-reported outcomes and adher- 
ence or quality of implementation of the school component. 
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Table 3 

Baseline and post-intervention means and standard deviation for outcome measures and covariate-adjusted regression results. 


Domain/measure 

Comparison (n : 

= 22) 


Tertiary FS (n = 33) 



Condition effect 

Effect size 


Baseline 

M(SD) 

Post-intervention 

M(SD) 

M Adj 

Baseline 

M(SD) 

Post-intervention 

M(SD) 

M Adj 

t 

p-Value 

Hedge’s g 

Pro-social behavior 

SSBD-ABI 

29.5(6.2) 

29.5(7.1) 

30.6 

32.3(5.8) 

39.2(9.7) 

38.4 

3.60 

<.001 

.89 

SSiS-SS-Teacher 

76.1 (8.8) 

75.4(9.2) 

76.0 

77.7 (9.7) 

91.7(14.3) 

89.8 

4.55 

<.001 

1.11 

SSiS-SS-Parent 

73.9(21.7) 

78.9 (21.0) 

80.1 

77.2(13.8) 

88.1 (17.4) 

87.0 

2.18 

.029 

.36 

Problem Behavior 

SSBD-MBI 

37.6(6.1) 

37.7(5.8) 

37.4 

36.6(5.7) 

28.1 (9.1) 

28.2 

-4.79 

<.001 

-1.17 

SSiS-PB-Teacher 

133.0(11.0) 

134.0(12.0) 

133.7 

132.4(11.8) 

118.5(16.2) 

119.4 

-4.15 

<.001 

-.98 

SSiS-PB-Parent 

129.1 (21.6) 

126.8(19.9) 

128.0 

129.9(10.1) 

116.7(12.8) 

115.6 

-3.39 

.001 

-.77 

Academic Competence 

SSiS-AC-Teacher 

89.3(17.9) 

88.0(16.9) 

87.9 

89.0(15.3) 

91.0(13.9) 

90.9 

1.54 

.125 

.19 


Notes: SSBD, Systematic Screening for Behavior Disorders; SSiS, Social Skills Improvement System Rating Scales; SS, Social Skills; MBI, Maladaptive Behavior Index; PB, 
Problem Behavior; AC, Academic Competence. 


For the home component, parent satisfaction was associated 
with improvements in parent-reported social skills on the SSiS 
(t=2.48, p = .013). For a one standard deviation increase in par- 
ent satisfaction at post-intervention, parent-reported social skills 
increased by .30 standard deviations. There were no statistically 
significant associations among parent satisfaction and the other 
home-based outcome measures. Coach-reported parent compli- 
ance was negatively associated with parent efficacy. For a one 
standard deviation increase in parent compliance, parent efficacy 
decreased by .40 standard deviations (t=-2.81, p = .005). Parent 
compliance was not significantly associated with the other out- 
come measures. There were no statistically significant associations 
among dose of the home component and the home-based outcome 
measures. 

Discussion 

Tertiary First Step provides a potentially effective, manualized 
intervention option for addressing the needs of an increasing num- 
ber of children who are entering school requiring tertiary level 
support for behavior problems (McCabe et al„ 2000; Walker et al., 
2004). The program meets the criteria for effective interventions 
to impact educational and mental health outcomes for children 
with challenging behavior provided by Hoagwood et al. (2007) and 
Patterson and Chamberlain (1994). Specifically, it includes a fam- 
ily component that is responsive to unique situations and needs 
of families requiring intensive, tertiary-level support, and provides 
flexible, yet manualized procedures that address parental motiva- 
tion to engage in the intervention process and facilitate change in 
their own behavior. 

This study builds on two decades of efficacy, effectiveness, and 
replication research related to the First Step intervention (Walker 
et ah, 2014) by providing preliminary evidence that the newly 
created variation for tertiary level students is promising for improv- 
ing important educational outcomes such as social skills, problem 
behavior, and academic engaged time. The consistency noted in 
outcomes across multiple domains, measures, and informants is 
a strength of this evaluation, and contributes substantially to our 
assertion that these First Step enhancements appear promising for 
improving educationally relevant outcomes. It is important to note 
this is the only First Step study in which the students’ behavioral 
impairments were identified in home and school settings prior 
to baseline assessments. The improvements in academic engaged 
time is perhaps the outcome that will be of the most interest to 
teachers and administrators and was the most powerful outcome 
associated with our within-subject analysis. Unfortunately, only a 
small effect size was observed for academic competence as assessed 
by the SSiS. 


Process data demonstrated the interventions were delivered 
with fidelity. Although this is not surprising for the school compo- 
nent, given the minor modifications to this aspect of the program 
at the tertiary level, it is noteworthy since this study repre- 
sents the first systematic application of the Tertiary homeBase 
intervention. Further, coaches, parents, and teachers all reported 
moderate to high levels of satisfaction with the revised home 
component. For the school component process-outcome analy- 
sis, dosage was associated with improvements in teacher-reported 
social skills and reductions in problem behavior, but adherence 
and quality were not. For the home component, parent satisfac- 
tion was associated with parent-reported social skills, but none 
of the other home-based outcomes. Given the small sample and 
high levels of adherence and implementation quality across tea- 
chers and coaches, we suspect that limited variability may be 
contributing to the lack of association among fidelity and out- 
come measures. Parent compliance and dosage were not associated 
with parenting efficacy or parental distress. Again, given the small 
sample size, these relationships must be interpreted with cau- 
tion. The analysis, however, does provide a model for examining 
the relationship between process variables and outcomes in future 
research. 

In addition to providing support for the notion that these 
enhancements to the First Step intervention are promising when 
implemented with children who require tertiary-level support, this 
study extends the literature to support the use of a motivational 
interviewing approach in school-based intervention research (Frey 
et al., 2011; Herman, Reinke, Frey, & Shepard, 2014; Reinke, Frey, 
Herman, & Thompson, 2014). This approach continues to appear 
worthy of future research. 

Limitations 

There are a number of limitations related to these findings. 
The primary limitation is that our design fails to control for sev- 
eral threats to internal validity. Specifically, while the addition 
of a comparison group increases confidence that the impressive 
gains in teacher- and parent-reported social skills and problem 
behaviors were the result of participation in the Tertiary First 
Step intervention, the lack of randomization limits our ability to 
rule out selection bias, and to some extent, history as possible 
threats to internal validity. Our sample size also prohibited us 
from conducting a post hoc analysis on the families that com- 
pleted two or fewer steps of Tertiary homeBase (N = 7), so that we 
might be able to identify patterns of families for whom we were 
unsuccessful at engaging. Additionally, while severity baseline 
scores for children in the intervention and comparison group were 
similar, more children in the intervention group were receiving 
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special education support through the school system. Additionally, 
observational data were not collected for the comparison group. 
Thus, history and maturation remain possible explanations for the 
changes observed from baseline to posttest within the intervention 
group. 

Our study design also prohibits us from isolating the specific 
benefits the Tertiary First Step program may have had over the 
original First Step intervention, or for identifying the relative con- 
tributions of the home and school components. We have no way of 
knowing if the effect sizes for our main outcomes would have been 
as large had we implemented the original intervention with the 
same sample of children. It is important to note that the teacher- 
reported effect sizes are slightly more robust than they have been 
in previous First Step studies employing similar designs (Walker 
et al., 1998, 2009). This finding is particularly impressive given that 
the sample recruited is substantially more at risk than has been the 
case in previous studies, and that previous finding have been more 
variable for children at the tertiary level. Our sample size is also 
relatively small. 

Next, there are a few limitations that constrain the external 
validity of the findings. For example, the coaches were all masters- 
level professionals who engaged in more training and supervision 
than could reasonably be expected to be available in educational 
settings under ordinary conditions. Additionally, our sample was 
drawn from only two school districts, which are not likely repre- 
sentative of districts generally. Finally, the intervention protocol 
changed slightly between the first and second cohort, although we 
do not view the changes as substantial. 

Future research 

Future research efforts should employ designs capable of estab- 
lishing the efficacy of the Tertiary First Step intervention by 
controlling for the threats to internal and external validity men- 
tioned above. Additionally, it will be important to determine the 
resources needed to train coaches who have not participated in the 
development of the intervention to implement it with fidelity. Next, 
it is important to add to the measurement protocol direct, sensitive 
measures of academic achievement. Further, because the Tertiary 
homeBase is intended to increase motivation to adopt effective par- 
enting practices, it would be beneficial to incorporate measures of 
motivation and direct observations of parenting practices to assess 
change following participation in the intervention. Future studies 
should also examine the impact of the intervention after the coach 
support has been withdrawn, and during the following school year 
when the child’s teacher and peer group have changed. Finally, it 
might also be to examine the relative contribution of the home and 
school components. 

Conclusion 

Few evidence-based interventions exist that are successful in 
altering the developmental pathways of young children with severe 
behavior problems who require tertiary level interventions. The 
enhancements to the First Step to Success early intervention 
program appear to be a promising option to address this need. 
Although the enhancements will require additional investigation 
and evaluation, the results from this initial implementation effort 
are encouraging, and suggest exposure to the intervention may 
lead to increases in social skills, decreases in problem behavior, 
and increases in academic engaged time. The evidence presented 
herein suggests the enhanced version of the First Step intervention 
is promising for expanding the reach of the program to children 
with tertiary-level needs. 


Author note 

An Institute of Education Sciences, US Department of Education 
grant (R324A090237) to the University of Louisville was utilized as 
partial support for the development of this manuscript. The opin- 
ions expressed are those of the authors and do not represent views 
of the Institute or the US Department of Education. We would 
also like to thank our partners in Jefferson County Public School 
(Louisville, KY) system. 

Appendix A. Supplementary data 

Supplementary data associated with this article can be 
found, in the online version, at http://dx.doi.Org/10.1016/j.ecresq. 
2014.05.002. 

References 

Abidin, R. R. (1995). Parenting Stress Index: Professional manual (3rd ed.). Odessa, FL: 
Psychological Assessment Resources, Inc. 

Achenbach, T. M. (1991). Manual for the Child Behavior Checklist 4-18 and the 1991 
profile. Burlington, VT: University of Vermont Department of Psychiatry. 
Batsche, G. M., Elliott, J., Graden, J. L., Grimes, J., Kovaleski, J. F., & Prasse, D. (2005). 
Response to intervention: Policy considerations and implementation. Alexandria, 
VA: National Association of State Directors of Special Education. 

Cohen, J. (1988). Statistical power analysis for the behavioral sciences. Hillsdale, NJ: 
Erlbaum. 

Collins, L. M., Schafer, J. L., & Kam, C.-M. (2001 ). A comparison of inclusive and 
restrictive strategies in modern missing data procedures. Psychological Methods, 
6, 330-351. http://dx.doi.org/10.1037/1082-989X.6A330 
Dishion.T.J., & Stormshak, E. A. (Eds.). (2007). Intervening in children’s lives: An ecolog- 
ical, family-centered approach to mental health care. Washington, DC: American 
Psychological Association. 

Enders, C. (2010). Applied missing data analysis. New York, NY: The Guilford Press. 

Feil, E., Frey, A. J., Walker, H. M., Seeley, J., Golly, A., & Small, J. (2014). The efficacy 
of a home-school intervention for preschoolers with challenging behaviors: A ran- 
domized controlled trial of Preschool First Step to Success. Manuscript submitted 
for publication. 

Forgatch, M. S., & Patterson, G. R. (1985). Therapist behavior as a detriment for client 
noncompliance: A paradox for the behavior modifier. Journal of Consulting and 
Clinical Psychology, 53, 846-851. http://dx.doi.Org/10.1037/0022-006X.53.6.846 
Frey, A. J., Cloud, R. N., Lee, J., Small, J. W., Seeley, J. R., Feil, E., et al. (2011). The 
promise of motivational interviewing in school mental health. School Mental 
Health, 3, 1-12. http://dx.doi.org/10.1007/sl2310-010-9048-z 
Frey, A. J., Lee, J., Small, J. W., Seeley, J. R., Walker, H. M., & Feil, E. G. (2013a). Transport- 
ing motivational interviewing to school settings to improve engagement and 
fidelity of Tier 2 interventions. Journal of Applied School Psychology, 29, 183-202. 
http://dx.d 0 i. 0 rg/l 0.1 080/1 5377903.201 3.778774 
Frey, A. J., Lee, J., Small, J. W., Seeley, J. R., Walker, H. M., & Feil, E. G. (2013b). 
The Motivational Interviewing Navigation Guide: A process for enhancing 
teachers’ motivation to adopt and implement school-based interventions. 
Advances in School Mental Health Promotion, 6, 157-173. http://dx.doi.org/ 
1 0.1 080/1 754730X.201 3.804334 

Frey, A. J., Small, J. W., Feil, E., Seeley, J. R., Walker, H. W., & Golly, A. (2013). The feasi- 
bility of First Step to Success with preschoolers. Children & Schools, 35, 171-186. 

http://dx.doi.org/10.1093/cs/cdt014 

Frey, A. J., Walker, H. M., Seeley, J. R., Lee, J., Small, J. W., Golly, A., et al. 
(2013c). Tertiary First Step to Success Resource Manual. Retrieved from 
http://www.firststeptosuccess.org/resources.html 
Frey, A. J., Walker, H. M., Seeley, J. R., Lee, J., Small, J. W., Golly, A., et al. (2013d). 
First Step to Success Classroom Check-up Resource Manual. Retrieved from 
http://www.firststeptosuccess.org/resources.html 
Glynn, L. H., & Moyers, T. B. (2010). Chasing change talk: The clinician’s role in 
evoking client language about change. Journal of Substance Abuse Treatment, 39, 
65-70. http://dx.d 0 i. 0 rg/l 0.1 01 6/j.jsat.201 0.03.01 2 
Gresham, F. M., & Elliott, S. N. (2008). Social Skills Improvement System Rating Scales. 
Minneapolis, MN: NCS Pearson. 

Herman, K., Reinke, W., Frey, A. J., & Shepard, S. (2014). Motivational interviewing in 
schools: Strategies to engage parents, teacher, and students. New York: Springer 
Publishing Company. 

Hoagwood, K. E., Olin, S. S., Kerker, B. D., Kratochwill, T. R., Crowe, M., & Saka, N. 
(2007). Empirically based school interventions targeted at academic and men- 
tal health functioning. Journal of Emotional and Behavioral Disorders, 15, 66-92. 
http://dx.d 0 i. 0 rg/l 0.1 1 77/106342660701 50020301 
Hops, H., & Walker, H. (1988). CLASS: Contingencies for Learning Academic and Social 
Skills. Seattle, WA: Educational Achievement Systems. 

Kegan, S. L. (1990). Readiness 2000: Rethinking rhetoric and responsibility. The Phi 
Delta Kappen, 72, 272-279. http://dx.doi.org/10.1177/0192513X11415354 
Lee, J., Frey, A. J., Seeley, J., Small, J., Walker, H. M., Golly, A, et al. (in press). Adapt- 
ing motivational interviewing to an early intervention addressing challenging 


A.J. Frey et al./ Early Childhood Research Quarterly 30 (2015) 1-11 


11 


behavior: Applications with teachers. In E. McNamera (Ed.), Motivational inter- 
viewing with children and young people: Issues and further applications. Ainsdale, 
Southport, Merseyside PR8, United Kingdom: Positive Behaviour Management. 
Loman, S., Rodriguez, B., & Horner, R. (2010). Sustainability of a targeted intervention 
package: First Step to Success in Oregon .Journal of Emotional and Behavioral 
Disorders, 18, 178-191. http://dx.doi.org/10.1177/1063426610362899 
McCabe, L. A., Hernandez, M., Lara, S. L., & Brooks-Gunn, J. (2000). Assessing 
preschoolers’ self-regulation in homes and classrooms: Lessons from the field. 
Behavioral Disorders, 26, 53-69. http://dx.doi.org/10.1037/a0015861 
Michie, S., van Stralen, M. M., & West, R. (201 1 ). The behaviour change wheel: A new 
method for characterising and designing behaviour change interventions. Imple- 
mentation Science, 6, 42. http://dx.doi.org/10.1186/1748-5908-6-42. Retrieved 
from http://www.implementationscience.eom/content/6/l /42 
Miller, W. R., & Rollnick, S. R. (2012). Motivational interviewing: Helping people change 
(Applications of motivational interviewing) (3rd ed.). New York, NY: The Guilford 
Press. 

Miller, W. R., Yahne, C. E., Moyers, T. B., Martinez, J., & Pirritano, M. 
(2004). A randomized trial of methods to help clinicians learn motiva- 
tional interviewing. Journal of Consulting & Clinical Psychology, 72, 1050-1062. 
http://dx.doi.Org/10.1037/0022-006x.72.6.1050 
Moyers, T. B., & Martin, T. (2006). Therapist influence on client language during 
motivational interviewing sessions. Journal of Substance Abuse Treatment, 30, 
245-251 . http://dx.d 0 i. 0 rg/l 0.1 037/a001 6830 
Muthen, L. K., & Muthen, B. 0. (1998-2010). Mplus user’s guide (6th ed.). Los Angeles, 
CA: Author. 

NCP Research. (2000). Oregon Healthy Start Evaluation Manual. Retrieved from 

http : / / www.npcresearch.com 

Nock, M. K., & Kazdin, A. E. (2005). Randomized controlled trial of a brief 
intervention for increasing participation in parent management training. 
Journal of Consulting and Clinical Psychology, 73, 872-879. http://dx.doi.org/ 
1 0.1 037/0022-006X.73.5.872 

Patterson, G. R., & Chamberlain, P. (1994). A functional analysis of resistance dur- 
ing parent training therapy. Clinical Psychology: Science and Practice, 1, 53-70. 
http://dx.doi. 0 rg/lO.l 1 1 1/j. 1468-2850.1 994.tb00006.x 
Patterson, G. R., & Forgatch, M. S. (1985). Therapist behavior as a detriment for client 
noncompliance: A paradox for the behavior modifier. Journal of Consulting and 
Clinical Psychology, 53, 846-851. http://dx.doi.Org/10.1037/0022-006X.53.6.846 
Patterson, G., Reid.J., & Dishion, T. (1992). Antisocial boys. Eugene, OR: Castalia Press. 
Reid, J. B. ( 1 993 ). Prevention of conduct disorder before and after school entry: Relat- 
ing interventions to developmental findings. Development and Psychopathology, 
5(1/2), 243-262. http://dx.doi.org/10.1017/S0954579400004375 
Reid, M. J., Webster-Stratton, C., & Hammond, M. (2003). Follow-up of children who 
received the Incredible Years intervention for oppositional defiant disorder: 
Maintenance and prediction of 2-year outcome. Behavior Therapy, 34, 471-491. 
http://dx.d 0 i. 0 rg/l 0. 1 01 6/S0005-7894(03)8003 1 -X 
Reinke, W., Frey, A. J., Herman, K., & Thompson, C. (2014). Improving engagement 
and implementation of interventions for children with behavior problems in 
home and school settings. In H. Walker, & F. Gresham (Eds.), Handbook of 
evidence-based practices for students having emotional and behavioral disorders 
(pp. 432-445). New York: Guilford Publishing, Inc. 

Rosnow, R. L., & Rosenthal, R. (2008). Assessing the effect size of outcome research. 
In A. M. Nezu, & C. M. Nezu (Eds.), Evidence-based outcome research: A practical 
guide to conducting randomized controlled trials for psychosocial interventions (pp. 
379-404). New York, NY: Oxford University Press. 

Schafer, J. L., & Graham, J. W. (2002). Missing data : Our view of the state of the art. Psy- 
chological Methods, 7, 147-177. http://dx.doi.Org/10.1037/1082-989X.7.2.147 
Seeley, J. R., Small, J. W., Walker, H. M., Feil, E. G., Severson, H. H., Golly, A. M., 
et al. (2009). Efficacy of the First Step to Success intervention for students 
with attention-deficit/hyperactivity disorder. School Mental Health, 1, 37-48. 
http://dx.doi.org/! 0. 1 007/sl 23 1 0-008-9003-4 


Severson, H. H., Walker, H. W., Hope-Doolittle, J., Kratochwill, T. R., & Gresham, F. M. 
(2007). Proactive, early screening to detect behaviorally at-risk students: Issues, 
approaches, emerging innovations, and professional practices .Journal of School 
Psychology, 45, 1 93-223. http:/ /dx.doi.org/1 0.1 01 6/j.jsp.2006.1 1 .003 
Smith, J. D., Dishion, T. J., Shaw, D. S., & Wilson, M. N. (2013). Indirect effects of 
fidelity to the Family Check-Up on changes in parenting and early childhood 
problem behaviors. Journal of Consulting and Clinical Psychology, 81, 962-974. 
http://dx.doi.org/10.1037/a0033950 

Smith, J. D., Handler, L., & Nash, M. R. (2010). Therapeutic assessment for pread- 
olescent boys with oppositional-defiant disorder: A replicated single-case 
time-series design. Psychological Assessment, 22, 593-602. http://dx.doi.org/ 
1 0.1 037/a001 9697 

Smith, J. D., Wolf, N. J., Handler, L., & Nash, M. R. (2009). Testing the effec- 
tiveness of family therapeutic assessment: A case study using a time-series 
design. Journal of Personality Assessment, 91, 518-536. http://dx.doi.org/ 
10.1080/00223890903228331 

Sprague, J., & Golly, A. (2013). BEST behavior: Building positive behavior support in 
schools (2nd ed). Longmont, CO: Sopris Learning. 

Sumi, W. C., Woodbridge, M. W., Javitz, H. S., Thornton, S. P., Wagner, M., Rous- 
pil, K., et al. (2013). Assessing the effectiveness of First Step to Success: 
Are short-term results the first step to long-term behavioral improvements? 
Journal of Emotional and Behavioral Disorders, 21, 66-78. http://dx.doi.org/ 
10.1177/1063426611429571 

Walker, H. M., Kavanagh, K., Stiller, B., Golly, A., Severson, H. H., & Feil, E. G. 
(1998). First Steps: An early intervention approach for preventing school 
antisocial behavior. Journal of Emotional and Behavioral Disorders, 6, 66-80. 

http://dx.doi.org/10.1177/106342669800600201 
Walker, H. W., Ramsey, E., & Gresham, F. M. (2004). Antisocial behavior in school: 
Evidenced-based practices (2nd ed.). Belmont, CA: Wadsworth/Thompson Learn- 
ing. 

Walker, H., Seeley, J., Small, J., Severson, H., Graham, B., Feil, E., et al. (2009). 
A randomized controlled trial of the First Step to Success early inter- 
vention: Demonstration of program efficacy outcomes in a diverse, urban 
school district. Journal of Emotional and Behavioral Disorders, 17, 197-212. 
http://dx.d 0 i. 0 rg/l 0.1 1 77/1063426609341 645 
Walker, H., & Severson, H. H. (1990). Systematic Screening for Behavior Disorders 
(SSBD): User’s guide and technical manual. Longmont, CO: Sopris West. 

Walker, H. M., Severson, H. H., Seeley, J. R., Feil, E. G., Small, J. W., Golly, S. R., et al. 
(2014). The evidence base of the First Step to Success Early Intervention for 
preventing emerging antisocial behavior patterns. In M. Walker, & F. M. Gresham 
(Eds.), Handbook of evidence-based practices for students having emotional and 
behavioral disorders (pp. 518-536). New York, NY: Guilford. 

Walker, H. M., Severson, H. H., Seeley, J. R., Small, J. W., Golly, A., Frey, A.J, et al. 
(2012). The evidence base of the First Step to Success Early Intervention for pre- 
venting emerging antisocial behavior patterns. Unpublished document. Retrieved 
from http://www.firststeptosuccess.org/publications.html 
Walker, H. M., Stiller, B., Golly, A., Kavanagh, K., Severson, H. H., & Feil, E. G. (1997). . 
First Step to Success: Helping young children overcome antisocial behavior (Vol. 6) 
Longmont, CO: Sopris West. 

Webster-Stratton, C., Reid, M. J., & Hammond, M. (2004). Treating children with 
early-onset conduct problems: Intervention outcomes for parent, child, and 
teacher training .Journal of Clinical Child and Adolescent Psychiatry, 33, 105-124. 

http://dx.doi.org/10.1207/S15374424JCCP3301_ll 
Weist, M. D., Lever, N. A., Bradshaw, C., & Owens, J. S. (2014). Further advancing the 
field of school mental health. In M. D. Weist, N. A. Lever, C. Bradshaw, & J. S. 
Owens (Eds.), Handbook of school mental health: Research, training, practice, and 
policy (2nd ed, Vol. 6, pp. 1-16). New York, NY: Springer. 

What Works Clearinghouse. (201 1 ). Procedures and standards handbook (Version 2. 1 ). 
Washington, DC: U.S. Department of Education, Institute of Education Sciences. 


